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Evaluating Workplace Education Program Effectiveness 

One of the key components of any training or educational program is the program evaluation 
process. This process measures program effectiveness and success in terms of program 
components and standards developed and identified by training or educational providers. Its 
purpose is to provide information about program impact and return on investment to funding 
sources, directors, coordinators, staff, and participants. In Colorado, all programs funded with 
Adult Education Act dollars have evaluative processes in place to measure curriculum and 
instruction; educational gains; recruitment; retention; staff development; and support services. 
These program components are measured against indicators and minimum program standards 
developed by educators in the field to ensure quality educational services. This system of 
evaluation is similar to the evaluation process used in a workplace education program because it 
is based on program standards that demonstrate overall program effectiveness using measurable 
outcomes 

Measurable outcomes are the cornerstone of workplace education program evaluation because 
they present vital information to business partners and provide for continuous program 
improvement. This process begins with the developmental stage of the workplace education 
program and continues throughout all program components. For example, the desired outcomes 
of the workplace education program will be identified in the initial program planning stages with 
input from the workplace education advisory council. Next, evaluative instruments will be 
developed and used throughout the program to provide feedback that allows for review and 
revision of program components. Finally, this process helps to ensure that the desired outcomes 
of the workplace education program are satisfied and directs the program developers in planning 
for program success. 

This guide will provide project directors, coordinators and professional staff an understanding of 
the workplace education program evaluation process; the differences and similarities between 
formative and summative evaluations; and appropriate evaluative instruments. Several 
supplemental resources are listed in the back of this guide for additional information on 
workplace education program evaluation. 



Training Evaluation in Business and Industry 

Just as adult educators strive to evaluate their educational programs, trainers in business and 
industry continue to develop new methods of evaluating company training. Companies evaluate 
training for many of the same reasons adult educators perform program evaluations; they must 
provide supporting data to their fimding source (the company), company managers, staff, and the 
training participants. Training evaluation is one of the key considerations for companies looking 
at training budgets, especially for those companies having to make budget cuts throughout the 
organization. After all, the costs of providing training are easier to see than the costs of not 
providing training. This is especially true when we talk about "soft skills" training. 
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Soft skills training involves those skills that are demonstrated through changed behaviors which 
result in changed performance on the job. They are those skills which are hard to detect or 
measure unless evaluators are aware of the varying behaviors to look for in observations. For 
example, teamwork skills are considered soft skills because although they may result in 
improved productivity, the actual skills are difficult to detect unless specific behaviors are 
identified and observations made. Because soft skills are less readily noticed in the workplace, 
training in these skills is often seen as unnecessary, and the costs involved may be considered 
excessive. This perception is important to workplace education providers because the basic skills 
taught in their programs are often considered to be soft skills. 

Because workplace education partnerships involve various partners, the evaluation process 
should be designed to meet the needs of all partners. This means that in addition to evaluating 
curriculum and instruction; educational gains; recruitment; retention; staff development; and 
support services, the workplace education partners will measure the training impact for the 
company. Before looking at the various methods of evaluation, it is important to know what 
businesses look for in outside training providers in terms of evaluation. 

Evaluation: What Business Wants 

As mentioned in the previous section, evaluation is an important consideration for in-house 
company trainers and outside educational providers who contract services with business. The 
data gathered through the program evaluation validates the need for training and verifies the 
impact of the training program in terms of measurable outcomes. There are many ways to 
evaluate training effectiveness. Most workplace education business partners will look to the 
educational provider to lead the design of the evaluation process; however, there will be basic 
considerations that most business partners will require of the educational partner in evaluating 
program effectiveness. These considerations include: 

• Evaluation involves key players (managers, supervisors, and employees). 

• Evaluation consists of multiple evaluation measures of quality or impact. 

• Evaluation incorporates continuous feedback.. 

• Evaluation is used to review and revise training as needed. 

• Evaluation assesses program impact using measurable outcomes. 

Workplace education providers will want to address each of these considerations during the 
initial conversations with business partners and incorporate them in the development of 
evaluation processes. 
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Forms of Evaluation 

Because evaluation occurs throughout the workplace education program, it is helpful for 
workplace education providers to understand what processes are appropriate to use at different 
stages of program development. The following will assist education providers in grouping 
evaluation processes to facilitate the development of the evaluation process. There are two 
general categories in which to group evaluation processes: formative evaluation and summative 
evaluation. 

Formative Evaluation 

As implicated by its name, formative evaluation involves those evaluation processes that are 
generally used during the beginning and middle stages of program operation. These processes 
assist in identifying problem areas that can be addressed and modified while change is possible 
and productive for the workplace education program. Frequently these evaluations are used to 
determine if all partners have shared goals and a common understanding of the program 
components, processes, definitions, and directions. Additionally, these evaluations are used to 
determine whether the learning processes and methods utilized are sufficient in accomplishing 
the goals of all involved in the workplace education program. There are many evaluation tools 
or instruments that are used when conducting formative evaluation. These include: 

• Supervisor Interviews 

• Employee Interviews 

• Supervisor Observations 

• Employee Self-observations 

• Document Analysis 

- needs assessments 

- curriculum 

- written evaluations 

• Classroom evaluations 

• Individual Education Plans 

• Anecdotal Information 

Each of these instruments or evaluation methods provides workplace education providers with 
valuable information that will assist in continuous program improvement while providing 
intermittent feedback to the workplace education partners. Formative evaluation also addresses 
at least four of the five key considerations described in the previous section: 1) they involve the 
key players; 2) there are multiple evaluation measures used; 3) they provide continuous 
feedback; and 4) they are used to review and revise the training as needed. 

Summative Evaluation 

In contrast to formative evaluation, summative evaluation usually takes place at the end of the 
program operation. It is designed to assess how well the workplace education program has 
succeeded through the use of pre and post program data. These evaluation measures usually 
assess the basic skills abilities of the students; the transfer of these skills to the job; the adult 
learner's beliefs about basic skills and education; and the transfer of training to the workplace in 
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terms of measurable outcomes identified in the initial stages of the workplace education program. 
The measurement instruments used in summative evaluation include the following: 

• Standardized Tests/Assessments 

• Customized Tests/Assessments 

• Supervisor Questionnaires (related to basic skills practices) 

• Employee Questionnaires (related to basic skills practices) 

• Pre and Post Supervisor Interviews 

• Pre and Post Employee Interviews 

• Subsequent Performance Ratings (identified as measurable outcomes) 

Obviously some of the formative and summative evaluation instruments can be used for a variety 
of purposes and are not exclusive to either category; however, these lists should provide 
education providers with a collection of instruments to pull from while designing and 
implementing the evaluation process. The next section of this guide will further delineate an 
evaluation process that assesses workplace-specific measurable outcomes. 

Evaluation: Outcomes Measurement 

Adult educators know that there are many factors that can influence learning. For example, 
individual students have various educational background experiences that influence learning; the 
learning environment impacts the adult learners experience; and program components must 
reflect good adult education practices to provide sound learning opportunities for adult learners. 
In business, trainers are also aware of the many factors that affect training outcomes. For 
example, management and supervisors must support training efforts for skills to transfer to the 
workplace; the company must provide for quality training to impact employees' learning 
experiences; and the company culture must encourage learning with words and actions. Just as 
there are many considerations that impact educational experiences, there are also many 
interrelated factors that come into play when trying to measure the impact of training in the 
workplace. 

It is difficult to ascertain the cause of improved quality, productivity, customers service, and 
other outcomes often used to measure the result of training programs because of the other 
influences that could impact these measures. For instance, the productivity of Department F 
increased after implementing a workplace education program targeted to that department. 
However; during the workplace education program. Department F implemented a new 
production process, and transferred in another supervisor. Can the workplace education partners 
state that it was the workplace education program that impacted productivity? These questions 
frequently arise when trying to demonstrate the workplace-specific measurable outcomes or 
return on investment (ROI) resulting from soft-skills training. Nevertheless, with proper 
resources, strong evidence can be collected to establish the returns of workplace education 
programs. 
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Level Four Evaluation 

Donald L. Kirkpatrick (1959) outlined a four level model for evaluating training that is 
commonly used in business. Each level of this model describes a measurement of training, each 
more complex than the preceding level. This model, as outline below, can be used to measure 
the impact of both technical skills and soft skills training; however, until recently, trainers 
generally stopped measuring soft skills training impact at Level Three because data required for 
Level Four evaluation was difficult to collect. This section will describe a process based upon 
Kirkpatrick's model that allows for Level Four evaluation in workplace education programs. 

In general terms, the four levels of evaluation outlined by Kirkpatrick include: 












Level One: 

Level Two: 
Level Three: 
Level Four: 



the measurement of the training participants' reactions to training at the 
time of training 

the measurement of learning the content of the training material 
the measurement of the use of new skills taught in training on the job 
the measurement of the observable business results or the return on 
investment as a result of training 



Most adult educators will recognize a semblance of these levels of evaluation to those measures 
used in adult education programs. For example. Level One evaluation in an adult education 
program may involve daily feedback collected from students in the form of interviews or 
anecdotal information; Level Two evaluation may involve comprehension checks used in 
conjunction with the adult education curriculum or assessment measures; and Level Three 
evaluation may include standardized skills assessments or customized assessments using realia. 
Level Four evaluation, however, is seldom used in adult education programs because it requires 
observations of the skills learned in the student's daily life with a measurable outcome that shows 
training impact on the student's daily life. This level of evaluation requires observation of the 
student after the training has occurred and is seldom possible for adult education programs to 
perform because students generally leave the program after the training is concluded. In contrast, 
workplace education providers have the opportunity to observe training impact on workers 
because the workers will remain in the company and measurable outcomes can be established for 
this evaluation process. 

Levels One and Two of Kirkpatrick's model involve those formative evaluation measures listed 
previously in this guide. They are those that allow for program change and development. Levels 
Three and Four, on the other hand, generally fall into the summative evaluation measures 
because they are those that measure the post-training impact. However, of these summative 
evaluation measures. Level Four requires the most in terms of time and resources and remits the 
most data in terms of training impact to the business. 



The following will provide the steps involved in conducting Level Four evaluation in a 
workplace education program. This planning process will assist workplace education providers 
during every step of the program planning stage. 
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Step One begins during the initial stages of program development with the 
workplace education advisory council identifying the critical job tasks to address 
in training. These critical job tasks reflect the outcomes that the company is 
looking for. For example, if the advisory council identifies the utilization of 
quality measurement documents as a critical job task, the workplace education 
provider will analyze this job task to teach the basic skills necessary in 
successfully using quality measurement documents. 

Step Two involves the Job Task Analysis. Once the critical job tasks have been 
identified, the workplace education provider will observe competent workers 
performing these tasks to analyze what skills are necessary in successfully 
performing each task on the job. This provides the basis for behavioral 
observation. The workplace education provider will want to note what behaviors 
are exhibited, and what skills are observed in an employee who performs the task 
correctly. These observations will be the measurable outcomes of Level Four 
evaluation. 

Step Three consists of a detailed analysis of the data gathered during the job task 
analysis to identify what will be taught in the workplace education program. This 
provides the structure for the development of the customized curriculum. 

Step Four requires the education provider to identify and/or develop skills 
assessments that measure the skills identified in the job task analysis and targeted 
in the customized curriculum. This step also requires all workplace education 
partners to identify how these skills will be measured on the job. For example, 
the partners may decide to compare the performance of employees who went 
through training with those who have not. This type of comparison can be valid if 
the two groups of employees demonstrated similar skills and performed similar 
tasks prior to instruction. This step is very important because it provides the 
measurable outcomes that will be used in evaluating program effectiveness. 

Step Five entails the development of the customized curriculum and lesson plans 
that will meet the needs of the students and facilitate the instructor in teaching 
those skills that were targeted to be included in instruction. 

Level Four evaluation follows the program planning steps outlined in each of the guides included 
in this Workplace Education Program Development Notebook. Although completed during the 
last stages of the workplace education program, this evaluation process incorporates all the 
planning stages and validates the success of the overall program in terms of the measurable 
outcomes outlined by the business partner. 
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All levels of evaluation outlined in Kirkpatrick's model should be used in evaluating program 
effectiveness. Both summative and formative evaluation have their place in providing quality 
instruction to employees and solid data to employers who search for reasons to support their 
decisions to provide for this training in their budgets. Evaluation provides the road map to 
program success. 



Kirkpatrick's four level model is applied to workplace education program evaluation and 
illustrated in this chart taken from The Bottom Line: Basic Skills in the Workplace (1988): 



Type/ Level 


Purpose 


Strengths 


Weaknesses 


Examples 


Student Reaction 


Measure student feelings 
about a program/course 


• Easy to administer 

• Provides immediate 
feedback on 
instructors, 
facilities, and 
program design 


• Subjective 

• Provides no 
measurement of 
learning, transfer of 
skills or benefit to 
the program 


• “Happiness” reports 

• Informal student/ 
instructor interview 

• Group Discussion 


Student Learning 


Measuring the amount of 
learning that has occurred 
in a program/course 


• Provides objective 
data on the 
effectiveness of 
training 

• Data can be 
collected before 
students leave the 
training program 


: • Requires skill in test 

construction 

• Provides no 

measurement of 
transfer of skills or 
benefit to the 
organization 


• Written pre/post 
tests 

• Skills laboratories 

• Role Plays 

• Simulations 

• Projects or 
presentations 

• Oral examinations 


Student Performance 


Measure the transfer of 
training 


• Provides objective 

data on impact to 
Job situation 


• Requires task 
analysis skills to 
construct and is 
time consuming to 
administer 

• Can be a 
’’politically” 
sensitive issue 


• Performance 
checklists 

• Performance 
appraisals 

• Critical checklists 
analysis 

• Selfappraisai 

• Observation 


Organization Results 


Measure impact of 
training on organization 


• Provides objective 

data for cost/benefit 
analysis and 
organizational 
support 


• Requires high level 
of evaluation design 
skills; requires 
collection of data 
over a period of 
time 

• Requires knowledge 
of organization 
needs and goals 


• Employee 
suggestions 

• Manufacturing 
indexes (cost, scrap, 
schedule 

compliance, quality, 

equipment 

donation) 

• Union grievances 

• Absenteeism rates 

• Accident rates 

• Customer 
complaints 
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General Guidelines to Evaluation 

There are basic guidelines to keep in mind while developing the workplace education evaluation 
process: 

• Keep it simple 

• Use economical evaluation methods 

• Tie the evaluation to company needs 

• Use both formative and summative evaluation methods 

• Adjust the workplace education program based on the results 

• Remain flexible 

• Be creative 

• Remember the bottom line 

• Publish and report the results in accordance to the confidentiality agreement 



This guide has focused on workplace education program evaluation; the difference and 
similarities between formative and summative evaluation; and identifying appropriate evaluative 
instruments for workplace education programs. For more information on workplace education 
program evaluation, please refer to the resources listed in the back of this guide. As always, the 
Office of Adult Education will provide technical assistance or training to facilitate your 
workplace education efforts. 



For assistance contact: Douglas Glynn 

Workplace Education Consultant 
(303) 866-6936 
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